Today, a dramatic increase is observed in the number of computer threats whose program code is executed by a virtual machine (for example, Java Virtual Machine, Common Language Runtime, ActionScript Virtual Machine). Exploits are the most dangerous of such threats. An exploit is a fragment of a program code or a sequence of commands which uses vulnerabilities in software and is used to attack a computer system. The danger lies not in the exploits themselves, but in the payload that they carry with them. The payload of an exploit is a functionality deployed by the offender, which, when a vulnerability is exploited on a system under attack, results in unauthorized activation of the functionality. Downloads of malicious software can be cited as an example of such functionality. Exploits can be used either independently, to test computer systems security, or together with malicious software.
From the wide variety of exploits, special note should be given to those that require a virtual machine to execute their code. This kind of exploits is most often used for attacks and is the most dangerous, as it is difficult to detect.
There are two main approaches for countering this type of threat. The first method involves elimination of the vulnerability used by the exploit. The second method involves using special tools for detecting the presence of exploits and stopping their activity. Such tools can be either built in the software itself (for example, the Java virtual machine security model) or provided externally. The first approach is reliable and addresses a root cause of the problem, but it has two significant drawbacks. For one, a rather long period of time passes from the moment the vulnerability is found to the moment the corrected software version is issued. The users of the vulnerable product remain unprotected throughout this period. Another drawback is that the first approach does not provide any protection from the so-called “zero day” vulnerabilities, i.e., threats that use an error or a vulnerability in the application or the operating system and arise immediately after the vulnerability is found, but before the relevant upgrade is issued.
The second approach avoids these drawbacks, but its reliability depends on the quality of its technical realization, and it should be noted that such protection tools can themselves be vulnerable. The most widespread solutions which use this approach are detection of exploits using heuristic rules and signature analysis (e.g., analysis of whether the analyzed code is identical to samples of the code of known computer threats), and built-in virtual machine security tools. The use of signatures is generally suitable for detection of known exploits. However, if the attacking code is modified, this solution will turn out to be useless.
The heuristic analysis implementation does not have this deficiency, but it can be inefficient in cases where there is a more elaborate code modification (e.g., encryption/obfuscation), where there is a change of the malicious code's algorithm, or where techniques to avoid code emulation are deployed.
A virtual machine is a software-based computing environment which runs on a hardware platform and operating system of a computer system. The virtual machine establishes a level of abstraction to achieve independence from the hardware platform on which the virtual machine is actually executed. Virtual machines have their own built-in security models. Special note should be given to the Java Virtual Machine (JVM) security model; which has four components: a class file verifier, a class loader, a security manager and the JVM architecture itself. Since Java byte code can be interpreted, it is possible to control an array indexes making it possible to avoid buffer overflows, which represent the most typical and dangerous type of software execution error. There are also built-in mechanisms for processing exceptions allow to efficiently solve arising conflicts, while a trash collector cleans unused memory, preventing the offender from viewing the “trash” memory blocks, which may contain useful information.
The security manager, the most important element in the security model for JVM, is a component which grants rights to applications, in accordance with the established security policy. If a situation arises where an application attempts to perform a privileged operation, the security manager checks the application's rights and determines the legitimacy of such behavior. The default security manager is the Java-class java.lang.SecurityManager; which includes several methods for checking operations critical to the security policy.
Recently, the number of directed attacks on JVM using exploits has dramatically increased. As these attacks have shown, the security model proposed by Java creators has, in practice, serious deficiencies in its implementation. These deficiencies are now actively used by offenders in their attacks.
In view of the above, known approaches for detection of exploits either have limitations in their application, or have deficiencies that create a security risk and generally do not provide adequate protection.
A practical solution is therefore needed that addresses at least some of these challenges, and that potentially, has even wider applicability.